Currently successful methods for video description are based onencoder-decoder sentence generation using recur-rent neural networks (RNNs).Recent work has shown the advantage of integrating temporal and/or spatialattention mechanisms into these models, in which the decoder net-work predictseach word in the description by selectively giving more weight to encodedfeatures from specific time frames (temporal attention) or to features fromspecific spatial regions (spatial attention). In this paper, we propose toexpand the attention model to selectively attend not just to specific times orspatial regions, but to specific modalities of input such as image features,motion features, and audio features. Our new modality-dependent attentionmechanism, which we call multimodal attention, provides a natural way to fusemultimodal information for video description. We evaluate our method on theYoutube2Text dataset, achieving results that are competitive with current stateof the art. More importantly, we demonstrate that our model incorporatingmultimodal attention as well as temporal attention significantly outperformsthe model that uses temporal attention alone.
展开▼
